{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入必要的库\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn import preprocessing\n",
    "from sklearn.metrics import mean_squared_error\n",
    "import seaborn as sns\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import StandardScaler,MinMaxScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPRegressor   #神经网络回归\n",
    "from sklearn.metrics import mean_absolute_error    #平均绝对误差MAE\n",
    "from sklearn.metrics import mean_squared_error    #平均平方误差MSE\n",
    "from sklearn.ensemble import RandomForestRegressor  #随机森林\n",
    "from xgboost import XGBRegressor                   #XGBoost\n",
    "from deepforest import CascadeForestRegressor      #深度森林\n",
    "import joblib                                      #保存模型\n",
    "plt.style.use('classic')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "根据2019年1月的估计，超过3.4万个尺寸大于10厘米的物体绕地球运行。其中22 300个被空间监视网跟踪，它们的位置以全球共享目录的形式公布。\n",
    "\n",
    "\n",
    "\n",
    "欧空局的空间碎片办公室支持欧空局的“Aeolus”、“Cryosat-2”和低地球轨道的“群- a /B/C”任务，以及接近地球静止区域的高度偏心轨道的“群- ii”任务。除此之外，还支持十几个合作机构和商业运营商的航天器。\n",
    "\n",
    "\n",
    "\n",
    "在这一活动的范围内，传播这些卫星的轨道，当探测到与目录内任何物体的近距离接近时，就汇编并发布一份联合数据电文。每个CDM包含多个关于方法的属性，例如所讨论的卫星的身份、潜在对撞机的对象类型、最接近的时间(TCA)、不确定性(即协方差)等。它还包含一个自我报告的风险，这是使用CDM中的一些属性计算出来的。在第一次CDM之后的日子里，随着物体位置的不确定性变小，其他的CDM被释放出来，精炼了近距离接触所获得的知识。\n",
    "\n",
    "\n",
    "\n",
    "通常，针对每一种独特的接近方法发布一组为期一周的CDMs时间序列，每天大约有3个CDMs可用。对于一个给定的接近方法，最后获得的清洁发展机制，包括计算的风险，可以假定是我们所拥有的关于潜在碰撞和两个物体的状态的最佳知识。大多数情况下，空间碎片办公室会在近距离接近前2天通知控制小组，并开始考虑潜在的规避策略，以避免碰撞的风险，然后在1天前做出最终决定。在这个挑战中，我们要求建立一个模型，利用最接近方法前2天记录的清洁发展机制来预测最终风险(即在接近方法前最后可用的清洁发展机制中预测的风险)。\n",
    "\n",
    "\n",
    "\n",
    "关于本次比赛中使用的数据集和各种CDMs中包含的属性的更多信息，可以在data部分找到。阅读这篇文章，你也可以了解到更多关于ESA空间碎片办公室处理避碰策略的当前方式。\n",
    "\n",
    "\n",
    "\n",
    "我们感谢美国空间监视网络为欧空局航天器安全运行提供的监视数据。具体来说，我们非常感谢这份协议，允许我们为本次比赛的目的公开发布数据集。轨道卫星之间主动避碰依靠有效、准确、及时的空间监视数据，已成为空间作业的常规任务。对于一颗位于近地轨道的典型卫星来说，每星期都会发出数百个警报，对应于卫星与另一空间物体可能发生的近距离接触(以连接数据消息CDMs的形式)。在自动处理和过滤后，每艘飞船和每周仍有大约2个可操作的警报，需要分析师进行详细的跟踪。平均而言，在欧洲航天局，每颗卫星每年都要执行一次以上的避碰操作。\n",
    "\n",
    "在这一挑战中，你的任务是建立一个模型来预测给定卫星与空间物体(如另一颗卫星、空间碎片等)之间的最终碰撞风险评估。为此，您将可以访问ESA精心准备的真实世界连接数据消息数据库。了解更多关于挑战和数据的信息。\n",
    "\n",
    "训练和测试数据之间的差异\n",
    "\n",
    "每个数据集由几个独特的事件(两个对象之间的亲密接触)组成，这些事件由event_id列中的唯一数字索引。\n",
    "\n",
    "\n",
    "\n",
    "训练集有162634行和13154个惟一事件(每次近距离接触平均给出大约12行/CDMs)。\n",
    "\n",
    "\n",
    "\n",
    "测试集有24484行和2167个惟一事件(每次近距离接触平均给出约11行/CDMs)。\n",
    "\n",
    "\n",
    "\n",
    "注意:测试集和训练集并不是随机从数据库中抽样的。换句话说，尽管它们来自相同的数据库，具有相同的收集过程和相同的特征，但它们都是人工挑选的，目的是为了过度代表高风险事件，并创建一个有趣的预测模型。这是这场比赛的一个特点，高风险事件很少，但代表了一个有用的预测模型的真正最终目标。\n",
    "\n",
    "\n",
    "\n",
    "特别是，测试数据与训练集相比有两个主要的不同之处:\n",
    "\n",
    "\n",
    "\n",
    "它只包含最新CDM在最接近时间(TCA)的1天内(time_to_tca < 1)的事件。这是因为，在某些情况下，最新可用的CDM距离最近的(已知)时间要几天。如果认为最接近的实际时间前7天计算的风险可以很好地近似TCA的风险，那就错了。此外，在最接近的时间前许多天预测风险对我们来说并不是很大的兴趣。另一方面，训练集是未经过滤的，您会发现许多情况下最新可用的CDMs离TCA很远。我们选择将这些冲突事件保留在训练集中，因为当涉及到从测试集中预测事件时，它们可能仍然有用。\n",
    "\n",
    "\n",
    "\n",
    "在TCA的2天内没有cdm可以学习。换句话说，离TCA最近的数据至少还要2天才能得到。这是因为，如挑战部分所述，在近距离接近前至少2天要计划一次潜在的避碰机动。与上面类似，训练集将包含所有案例，包括在最接近之前至少2天没有数据可用的事件(即所有cdm在TCA的2天内的事件仍然存在于数据集中)。\n",
    "\n",
    "\n",
    "\n",
    "列描述\n",
    "\n",
    "数据集表示为一个表，其中每一行对应一个CDM，每个CDM包含103个记录的特征/特征。因此有103列，我们将在下面描述。数据集由几个唯一的碰撞/接近事件组成，它们在event_id列中标识。反过来，每次碰撞事件都是由数个随时间记录下来的cdm组成的。因此，单个碰撞事件可以看作是一系列cdm的时间序列。**从这些CDM中，对于每个碰撞事件，我们感兴趣的是预测在时间序列的最后一个CDM中计算的最终风险(即每个碰撞事件最后一行中的风险值)。**\n",
    "\n",
    "\n",
    "\n",
    "列描述,我们首先描述列具有惟一名称,然后**列名字的区别只取决于他们是否指的是目标对象(如果列名始于一个t)或螺纹梳刀对象(如果列名始于一个c)。在这里,目标指的是欧洲航天局卫星而螺纹梳刀指的是空间碎片/对象我们想要避免的。描述追逐者和目标共享的列名，我们用占位符x替换t和c。例如，c_sigma_r和t_sigma_r都对应于x_sigma_r的描述。**\n",
    "\n",
    "\n",
    "\n",
    "注意，**除了c_object_type之外，所有列都是数字的。**\n",
    "\n",
    "\n",
    "\n",
    "惟一的名称列\n",
    "\n",
    "风险:在每个CDM的纪元自我计算的价值[以10为底的日志]。在测试集中，这个值是要预测的，在每个event_id最接近的时候。请注意，如上所述，在测试集中，我们不知道CDMs中包含的最接近的2天内的实际数据，因为它们发生在“未来”。\n",
    "\n",
    "\n",
    "\n",
    "event_id:每个碰撞事件的唯一id\n",
    "\n",
    "time_to_tca: CDM创建和最接近时间之间的时间间隔[天]\n",
    "\n",
    "mission_id:受影响任务的标识符\n",
    "\n",
    "max_risk_estimate:通过缩放组合协方差得到的最大碰撞概率\n",
    "\n",
    "max_risk_scaling:用于计算最大碰撞概率的缩放因子\n",
    "\n",
    "脱靶距离:tca中追逐者与目标的相对位置[m]\n",
    "\n",
    "relative_speed:追击者与目标在tca上的相对速度[m/s]\n",
    "\n",
    "relative_position_n:追逐者与目标的相对位置:正常(交叉轨迹)[m]\n",
    "\n",
    "relative_position_r:追逐者与目标之间的相对位置:径向[m]\n",
    "\n",
    "relative_position_t:追逐者与目标之间的相对位置:横向(沿轨道)[m]\n",
    "\n",
    "relative_velocity_n:追逐者与目标之间的相对速度:正常(交叉轨迹)[m/s]\n",
    "\n",
    "relative_velocity_r:追逐者与目标之间的相对速度:径向[m/s]\n",
    "        \n",
    "relative_velocity_t:追逐者与目标之间的相对速度:横向(沿轨道)[m/s]\n",
    "\n",
    "c_object_type:与卫星存在碰撞风险的对象类型\n",
    "\n",
    "geocentric_latitude:合点的纬度[deg]\n",
    "\n",
    "方位角:相对速度矢量:方位角[deg]\n",
    "\n",
    "仰角:相对速度矢量:仰角[度]\n",
    "\n",
    "F10: 10.7 cm射电通量指数[10−22 W/(m2 Hz)]\n",
    "\n",
    "每日行星地磁振幅指数\n",
    "\n",
    "F3M: F10.7(超过3个太阳旋转)81天运行平均值[10 - 22 W/(m2 Hz)]\n",
    "\n",
    "SSN:狼号太阳黑子号\n",
    "\n",
    "追逐者和目标对象之间的共享列名\n",
    "\n",
    "x_sigma_rdot:协方差;径向速度标准差(sigma) [m/s]\n",
    "\n",
    "x_sigma_n:协方差;(交叉轨迹)位置标准差(sigma) [m]\n",
    "\n",
    "x_cn_r:协方差;法向(交叉轨道)位置与径向位置的相关性\n",
    "\n",
    "x_cn_t:协方差;法向(交叉轨迹)位置与横向(沿轨迹)位置的相关性\n",
    "\n",
    "x_cndot_n:协方差;法向(交叉轨迹)速度与法向(交叉轨迹)位置的相关性\n",
    "\n",
    "x_sigma_ndot:协方差;正(横向)速度标准差[m/s]\n",
    "\n",
    "x_cndot_r:协方差;法向(横向)速度与径向位置的相关性\n",
    "\n",
    "x_cndot_rdot:协方差;法向(横向)速度与径向速度的相关性\n",
    "\n",
    "x_cndot_t:协方差;法向(横向)速度与横向(横向)位置的相关关系\n",
    "\n",
    "x_cndot_tdot:协方差;法向(横向)速度与横向(横向)速度的相关关系\n",
    "\n",
    "x_sigma_r:协方差;径向位置标准差[m]\n",
    "\n",
    "x_ct_r:协方差;横向(沿轨道)位置与径向位置的相关性\n",
    "\n",
    "x_sigma_t:协方差;横向(沿轨)位置标准差(sigma) [m]\n",
    "\n",
    "x_ctdot_n:协方差;横向(沿轨)速度与法向(横轨)位置的相关性\n",
    "\n",
    "x_crdot_n:协方差;径向速度与法向(交叉轨道)位置的相关性\n",
    "\n",
    "x_crdot_t:协方差;径向速度与横向(沿轨道)位置的相关关系\n",
    "\n",
    "x_crdot_r:协方差;径向速度与径向位置的相关性\n",
    "\n",
    "x_ctdot_r:协方差;横向(沿轨道)速度与径向位置的相关关系\n",
    "\n",
    "x_ctdot_rdot:协方差;横向(沿轨道)速度与径向速度的相关关系\n",
    "    \n",
    "x_ctdot_t:协方差;横向(沿轨)速度与横向(沿轨)位置的相关关系\n",
    "\n",
    "x_sigma_tdot:协方差;横向(沿轨)速度标准差[m/s]\n",
    "\n",
    "x_position_covariance_det:协方差的行列式(~体积)\n",
    "\n",
    "x_cd_area_over_mass:弹道系数[m2/kg]\n",
    "\n",
    "x_cr_area_over_mass:太阳辐射系数。A/m(弹道系数等效)\n",
    "\n",
    "x_h_apo:远地点(-地球)[km]\n",
    "\n",
    "x_h_per:近地点(-Rearth)(公里)\n",
    "\n",
    "x_ecc:偏心\n",
    "\n",
    "x_j2k_inc:倾向(度)\n",
    "\n",
    "x_j2k_sma:半长轴[km]\n",
    "\n",
    "x_sedr:能量耗散率[W/kg]\n",
    "\n",
    "x_span:碰撞风险计算算法使用的尺寸(追逐者最小直径假设为2米)[m]\n",
    "\n",
    "x_rcs_estimate:雷达截面积[m2]\n",
    "\n",
    "x_actual_od_span:轨道确定更新间隔的实际长度[天]\n",
    "\n",
    "x_obs_available:可用于轨道确定的观测数(每CDM)\n",
    "\n",
    "x_obs_used:用于轨道确定的观测数(每CDM)\n",
    "\n",
    "x_recommend_od_span:轨道确定的建议更新间隔长度[天]\n",
    "\n",
    "x_residuals_accepted:轨道判定残差\n",
    "\n",
    "x_time_lastob_end:用于轨道确定的最后一次被接受的观测的时间间隔的结束日(相对于CDM创建纪元)\n",
    "\n",
    "x_time_lastob_start:用于确定轨道的最后一次被接受的观测的开始时间(以天为单位的CDM创建纪元)\n",
    "\n",
    "x_weighted_rms:最小二乘轨道确定的均方根"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.数据清洗"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>time_to_tca</th>\n",
       "      <th>mission_id</th>\n",
       "      <th>risk</th>\n",
       "      <th>max_risk_estimate</th>\n",
       "      <th>max_risk_scaling</th>\n",
       "      <th>miss_distance</th>\n",
       "      <th>relative_speed</th>\n",
       "      <th>relative_position_r</th>\n",
       "      <th>relative_position_t</th>\n",
       "      <th>relative_position_n</th>\n",
       "      <th>...</th>\n",
       "      <th>t_sigma_rdot</th>\n",
       "      <th>c_sigma_rdot</th>\n",
       "      <th>t_sigma_tdot</th>\n",
       "      <th>c_sigma_tdot</th>\n",
       "      <th>t_sigma_ndot</th>\n",
       "      <th>c_sigma_ndot</th>\n",
       "      <th>F10</th>\n",
       "      <th>F3M</th>\n",
       "      <th>SSN</th>\n",
       "      <th>AP</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.566798</td>\n",
       "      <td>5</td>\n",
       "      <td>-10.204955</td>\n",
       "      <td>-7.834756</td>\n",
       "      <td>8.602101</td>\n",
       "      <td>14923.0</td>\n",
       "      <td>13792.0</td>\n",
       "      <td>453.8</td>\n",
       "      <td>5976.6</td>\n",
       "      <td>-13666.8</td>\n",
       "      <td>...</td>\n",
       "      <td>0.147350</td>\n",
       "      <td>58.272095</td>\n",
       "      <td>0.004092</td>\n",
       "      <td>0.165044</td>\n",
       "      <td>0.002987</td>\n",
       "      <td>0.386462</td>\n",
       "      <td>89.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.207494</td>\n",
       "      <td>5</td>\n",
       "      <td>-10.355758</td>\n",
       "      <td>-7.848937</td>\n",
       "      <td>8.956374</td>\n",
       "      <td>14544.0</td>\n",
       "      <td>13792.0</td>\n",
       "      <td>474.3</td>\n",
       "      <td>5821.2</td>\n",
       "      <td>-13319.8</td>\n",
       "      <td>...</td>\n",
       "      <td>0.059672</td>\n",
       "      <td>57.966413</td>\n",
       "      <td>0.003753</td>\n",
       "      <td>0.164383</td>\n",
       "      <td>0.002933</td>\n",
       "      <td>0.386393</td>\n",
       "      <td>89.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.952193</td>\n",
       "      <td>5</td>\n",
       "      <td>-10.345631</td>\n",
       "      <td>-7.847406</td>\n",
       "      <td>8.932195</td>\n",
       "      <td>14475.0</td>\n",
       "      <td>13792.0</td>\n",
       "      <td>474.6</td>\n",
       "      <td>5796.2</td>\n",
       "      <td>-13256.1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.039258</td>\n",
       "      <td>57.907599</td>\n",
       "      <td>0.003576</td>\n",
       "      <td>0.164352</td>\n",
       "      <td>0.002967</td>\n",
       "      <td>0.386381</td>\n",
       "      <td>89.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>42.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.579669</td>\n",
       "      <td>5</td>\n",
       "      <td>-10.337809</td>\n",
       "      <td>-7.845880</td>\n",
       "      <td>8.913444</td>\n",
       "      <td>14579.0</td>\n",
       "      <td>13792.0</td>\n",
       "      <td>472.7</td>\n",
       "      <td>5838.9</td>\n",
       "      <td>-13350.7</td>\n",
       "      <td>...</td>\n",
       "      <td>0.022066</td>\n",
       "      <td>57.993905</td>\n",
       "      <td>0.003298</td>\n",
       "      <td>0.164309</td>\n",
       "      <td>0.002918</td>\n",
       "      <td>0.386400</td>\n",
       "      <td>89.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.257806</td>\n",
       "      <td>5</td>\n",
       "      <td>-10.391260</td>\n",
       "      <td>-7.852942</td>\n",
       "      <td>9.036838</td>\n",
       "      <td>14510.0</td>\n",
       "      <td>13792.0</td>\n",
       "      <td>478.7</td>\n",
       "      <td>5811.1</td>\n",
       "      <td>-13288.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.015075</td>\n",
       "      <td>57.946717</td>\n",
       "      <td>0.003670</td>\n",
       "      <td>0.164172</td>\n",
       "      <td>0.003220</td>\n",
       "      <td>0.386388</td>\n",
       "      <td>89.0</td>\n",
       "      <td>83.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 101 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   time_to_tca  mission_id       risk  max_risk_estimate  max_risk_scaling  \\\n",
       "0     1.566798           5 -10.204955          -7.834756          8.602101   \n",
       "1     1.207494           5 -10.355758          -7.848937          8.956374   \n",
       "2     0.952193           5 -10.345631          -7.847406          8.932195   \n",
       "3     0.579669           5 -10.337809          -7.845880          8.913444   \n",
       "4     0.257806           5 -10.391260          -7.852942          9.036838   \n",
       "\n",
       "   miss_distance  relative_speed  relative_position_r  relative_position_t  \\\n",
       "0        14923.0         13792.0                453.8               5976.6   \n",
       "1        14544.0         13792.0                474.3               5821.2   \n",
       "2        14475.0         13792.0                474.6               5796.2   \n",
       "3        14579.0         13792.0                472.7               5838.9   \n",
       "4        14510.0         13792.0                478.7               5811.1   \n",
       "\n",
       "   relative_position_n  ...  t_sigma_rdot  c_sigma_rdot  t_sigma_tdot  \\\n",
       "0             -13666.8  ...      0.147350     58.272095      0.004092   \n",
       "1             -13319.8  ...      0.059672     57.966413      0.003753   \n",
       "2             -13256.1  ...      0.039258     57.907599      0.003576   \n",
       "3             -13350.7  ...      0.022066     57.993905      0.003298   \n",
       "4             -13288.0  ...      0.015075     57.946717      0.003670   \n",
       "\n",
       "   c_sigma_tdot  t_sigma_ndot  c_sigma_ndot   F10   F3M   SSN    AP  \n",
       "0      0.165044      0.002987      0.386462  89.0  83.0  42.0  11.0  \n",
       "1      0.164383      0.002933      0.386393  89.0  83.0  42.0  11.0  \n",
       "2      0.164352      0.002967      0.386381  89.0  83.0  42.0  11.0  \n",
       "3      0.164309      0.002918      0.386400  89.0  83.0  40.0  14.0  \n",
       "4      0.164172      0.003220      0.386388  89.0  83.0  40.0  14.0  \n",
       "\n",
       "[5 rows x 101 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_path=r'E:\\yandex_down\\train_data\\train_data.csv'\n",
    "data=pd.read_csv(file_path,parse_dates=True)\n",
    "data.drop(columns=['c_rcs_estimate','event_id'],inplace=True)   #event_id只记录第几次事件，无效信息。c_rcs_estimate缺失五万多条数据，占比32.49%，所以删去（其余最多缺失九千多条）\n",
    "key=data.keys()\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 162634 entries, 0 to 162633\n",
      "Columns: 101 entries, time_to_tca to AP\n",
      "dtypes: float64(97), int64(3), object(1)\n",
      "memory usage: 125.3+ MB\n"
     ]
    }
   ],
   "source": [
    "data.info()     #有100列数值标签 1列字符串标签，需讲字符标签转换为数值标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从数据中划分出预测标签列\n",
    "y = data.risk\n",
    "X = data.drop(['risk'], axis=1)     #预测标签列'risk'\n",
    "\n",
    "# 将数据划分为训练集和测试集\n",
    "X_train_full, X_valid_full, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2,\n",
    "                                                               random_state=0)   #固定随机数种子\n",
    "\n",
    "#选取标签列，独立字符串标签数小于10且dtype格式为object\n",
    "categorical_cols = [cname for cname in X_train_full.columns if X_train_full[cname].nunique() < 10 and \n",
    "                        X_train_full[cname].dtype == \"object\"]\n",
    "# 选择数值列\n",
    "numerical_cols = [cname for cname in X_train_full.columns if X_train_full[cname].dtype in ['int64', 'float64']]\n",
    "\n",
    "# 只保留数值列和标签列\n",
    "my_cols = categorical_cols + numerical_cols\n",
    "X_train = X_train_full[my_cols].copy()\n",
    "X_valid = X_valid_full[my_cols].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#处理数值类型数据\n",
    "# numerical_transformer = SimpleImputer(strategy='constant')\n",
    "numerical_transformer =Pipeline(steps=[('imputer',SimpleImputer(strategy='most_frequent')),    #缺失值处理，用众数填充\n",
    "#                                       ('reduce_dim', PCA()),                                    #PCA降维，已经放弃，加了之后效果更差\n",
    "                                        ('scaler', StandardScaler())])               #这里用zscore标准化方法，即x'=(x-u)/sigma,也可以考虑用MinMaxScaler，即0-1归一化\n",
    "#处理字符串类型数据\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='most_frequent')),               #缺失值处理，用众数填充\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))                  #忽略未知的分类特征\n",
    "])\n",
    "#捆绑数值和分类数据的预处理\n",
    "preprocessor = ColumnTransformer(transformers=[('num', numerical_transformer, numerical_cols),\n",
    "        ('cat', categorical_transformer, categorical_cols)])\n",
    "preprocessor.fit(X_train)\n",
    "X_train=preprocessor.transform(X_train)\n",
    "X_valid =preprocessor.transform(X_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本文用到了许多sklearn中库的函数，详情请见 https://scikitlearn.com.cn/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.随机森林模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1随机森林模型1-100棵树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "building tree 1 of 100\n",
      "building tree 2 of 100building tree 3 of 100\n",
      "\n",
      "building tree 4 of 100\n",
      "building tree 5 of 100\n",
      "building tree 6 of 100\n",
      "building tree 7 of 100\n",
      "building tree 8 of 100\n",
      "building tree 9 of 100\n",
      "building tree 10 of 100\n",
      "building tree 11 of 100\n",
      "building tree 12 of 100\n",
      "building tree 13 of 100\n",
      "building tree 14 of 100\n",
      "building tree 15 of 100\n",
      "building tree 16 of 100\n",
      "building tree 17 of 100\n",
      "building tree 18 of 100\n",
      "building tree 19 of 100\n",
      "building tree 20 of 100\n",
      "building tree 21 of 100\n",
      "building tree 22 of 100\n",
      "building tree 23 of 100\n",
      "building tree 24 of 100\n",
      "building tree 25 of 100\n",
      "building tree 26 of 100\n",
      "building tree 27 of 100\n",
      "building tree 28 of 100\n",
      "building tree 29 of 100\n",
      "building tree 30 of 100\n",
      "building tree 31 of 100\n",
      "building tree 32 of 100\n",
      "building tree 33 of 100\n",
      "building tree 34 of 100\n",
      "building tree 35 of 100\n",
      "building tree 36 of 100\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  33 tasks      | elapsed:  1.7min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "building tree 37 of 100\n",
      "building tree 38 of 100\n",
      "building tree 39 of 100\n",
      "building tree 40 of 100\n",
      "building tree 41 of 100\n",
      "building tree 42 of 100\n",
      "building tree 43 of 100\n",
      "building tree 44 of 100\n",
      "building tree 45 of 100\n",
      "building tree 46 of 100\n",
      "building tree 47 of 100\n",
      "building tree 48 of 100\n",
      "building tree 49 of 100\n",
      "building tree 50 of 100\n",
      "building tree 51 of 100\n",
      "building tree 52 of 100\n",
      "building tree 53 of 100\n",
      "building tree 54 of 100\n",
      "building tree 55 of 100\n",
      "building tree 56 of 100\n",
      "building tree 57 of 100\n",
      "building tree 58 of 100\n",
      "building tree 59 of 100\n",
      "building tree 60 of 100\n",
      "building tree 61 of 100\n",
      "building tree 62 of 100\n",
      "building tree 63 of 100\n",
      "building tree 64 of 100\n",
      "building tree 65 of 100\n",
      "building tree 66 of 100\n",
      "building tree 67 of 100\n",
      "building tree 68 of 100\n",
      "building tree 69 of 100\n",
      "building tree 70 of 100\n",
      "building tree 71 of 100\n",
      "building tree 72 of 100\n",
      "building tree 73 of 100\n",
      "building tree 74 of 100\n",
      "building tree 75 of 100\n",
      "building tree 76 of 100\n",
      "building tree 77 of 100\n",
      "building tree 78 of 100\n",
      "building tree 79 of 100\n",
      "building tree 80 of 100\n",
      "building tree 81 of 100\n",
      "building tree 82 of 100\n",
      "building tree 83 of 100\n",
      "building tree 84 of 100\n",
      "building tree 85 of 100\n",
      "building tree 86 of 100\n",
      "building tree 87 of 100\n",
      "building tree 88 of 100\n",
      "building tree 89 of 100\n",
      "building tree 90 of 100\n",
      "building tree 91 of 100\n",
      "building tree 92 of 100\n",
      "building tree 93 of 100\n",
      "building tree 94 of 100\n",
      "building tree 95 of 100\n",
      "building tree 96 of 100\n",
      "building tree 97 of 100\n",
      "building tree 98 of 100\n",
      "building tree 99 of 100\n",
      "building tree 100 of 100\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed:  4.7min finished\n",
      "[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  33 tasks      | elapsed:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 0.012029525570638562\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed:    0.2s finished\n"
     ]
    }
   ],
   "source": [
    "#随机森林模型 100个树\n",
    "# 重要参数 n_estimators 弱学习器个数，默认100   oob_score  是否袋外采样 默认false  建议True\n",
    "#criterion 评价标准 回归RF有mae，默认mse 不用改\n",
    "# model = RandomForestRegressor(n_estimators=100, random_state=0,verbose=2)\n",
    "model2_1 = RandomForestRegressor(n_estimators=100,oob_score=True,verbose=2,n_jobs=4) #100棵树 开启袋外采样  打印状态  4个核心并行\n",
    "model2_1.fit(X_train,y_train)                    #训练集拟合\n",
    "preds2_1 = model2_1.predict(X_valid)             #进行预测\n",
    "# score2_1 = mean_absolute_error(y_valid, preds) #测量平均绝对误差MAE\n",
    "score2_1 = mean_squared_error(y_valid, preds2_1) #测量均方误差MSE\n",
    "# print('MAE:', score2_1) #0.034816310036587715\n",
    "print('MSE:', score2_1) #0.012138822486107472"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2随机森林模型2-200棵树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "building tree 1 of 200building tree 2 of 200\n",
      "\n",
      "building tree 3 of 200\n",
      "building tree 4 of 200\n",
      "building tree 5 of 200\n",
      "building tree 6 of 200\n",
      "building tree 7 of 200\n",
      "building tree 8 of 200\n",
      "building tree 9 of 200\n",
      "building tree 10 of 200\n",
      "building tree 11 of 200\n",
      "building tree 12 of 200\n",
      "building tree 13 of 200\n",
      "building tree 14 of 200\n",
      "building tree 15 of 200\n",
      "building tree 16 of 200\n",
      "building tree 17 of 200\n",
      "building tree 18 of 200\n",
      "building tree 19 of 200\n",
      "building tree 20 of 200\n",
      "building tree 21 of 200\n",
      "building tree 22 of 200\n",
      "building tree 23 of 200\n",
      "building tree 24 of 200\n",
      "building tree 25 of 200\n",
      "building tree 26 of 200\n",
      "building tree 27 of 200\n",
      "building tree 28 of 200\n",
      "building tree 29 of 200\n",
      "building tree 30 of 200\n",
      "building tree 31 of 200\n",
      "building tree 32 of 200\n",
      "building tree 33 of 200\n",
      "building tree 34 of 200\n",
      "building tree 35 of 200\n",
      "building tree 36 of 200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  33 tasks      | elapsed:  1.7min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "building tree 37 of 200\n",
      "building tree 38 of 200\n",
      "building tree 39 of 200\n",
      "building tree 40 of 200\n",
      "building tree 41 of 200\n",
      "building tree 42 of 200\n",
      "building tree 43 of 200\n",
      "building tree 44 of 200\n",
      "building tree 45 of 200\n",
      "building tree 46 of 200\n",
      "building tree 47 of 200\n",
      "building tree 48 of 200\n",
      "building tree 49 of 200\n",
      "building tree 50 of 200\n",
      "building tree 51 of 200\n",
      "building tree 52 of 200\n",
      "building tree 53 of 200\n",
      "building tree 54 of 200\n",
      "building tree 55 of 200\n",
      "building tree 56 of 200\n",
      "building tree 57 of 200\n",
      "building tree 58 of 200\n",
      "building tree 59 of 200\n",
      "building tree 60 of 200\n",
      "building tree 61 of 200\n",
      "building tree 62 of 200\n",
      "building tree 63 of 200\n",
      "building tree 64 of 200\n",
      "building tree 65 of 200\n",
      "building tree 66 of 200\n",
      "building tree 67 of 200\n",
      "building tree 68 of 200\n",
      "building tree 69 of 200\n",
      "building tree 70 of 200\n",
      "building tree 71 of 200\n",
      "building tree 72 of 200\n",
      "building tree 73 of 200\n",
      "building tree 74 of 200\n",
      "building tree 75 of 200\n",
      "building tree 76 of 200\n",
      "building tree 77 of 200\n",
      "building tree 78 of 200\n",
      "building tree 79 of 200\n",
      "building tree 80 of 200\n",
      "building tree 81 of 200\n",
      "building tree 82 of 200\n",
      "building tree 83 of 200\n",
      "building tree 84 of 200\n",
      "building tree 85 of 200\n",
      "building tree 86 of 200\n",
      "building tree 87 of 200\n",
      "building tree 88 of 200\n",
      "building tree 89 of 200\n",
      "building tree 90 of 200\n",
      "building tree 91 of 200\n",
      "building tree 92 of 200\n",
      "building tree 93 of 200\n",
      "building tree 94 of 200\n",
      "building tree 95 of 200\n",
      "building tree 96 of 200\n",
      "building tree 97 of 200\n",
      "building tree 98 of 200\n",
      "building tree 99 of 200\n",
      "building tree 100 of 200\n",
      "building tree 101 of 200\n",
      "building tree 102 of 200\n",
      "building tree 103 of 200\n",
      "building tree 104 of 200\n",
      "building tree 105 of 200\n",
      "building tree 106 of 200\n",
      "building tree 107 of 200\n",
      "building tree 108 of 200\n",
      "building tree 109 of 200\n",
      "building tree 110 of 200\n",
      "building tree 111 of 200\n",
      "building tree 112 of 200\n",
      "building tree 113 of 200\n",
      "building tree 114 of 200\n",
      "building tree 115 of 200\n",
      "building tree 116 of 200\n",
      "building tree 117 of 200\n",
      "building tree 118 of 200\n",
      "building tree 119 of 200\n",
      "building tree 120 of 200\n",
      "building tree 121 of 200\n",
      "building tree 122 of 200\n",
      "building tree 123 of 200\n",
      "building tree 124 of 200\n",
      "building tree 125 of 200\n",
      "building tree 126 of 200\n",
      "building tree 127 of 200\n",
      "building tree 128 of 200\n",
      "building tree 129 of 200\n",
      "building tree 130 of 200\n",
      "building tree 131 of 200\n",
      "building tree 132 of 200\n",
      "building tree 133 of 200\n",
      "building tree 134 of 200\n",
      "building tree 135 of 200\n",
      "building tree 136 of 200\n",
      "building tree 137 of 200\n",
      "building tree 138 of 200\n",
      "building tree 139 of 200\n",
      "building tree 140 of 200\n",
      "building tree 141 of 200\n",
      "building tree 142 of 200\n",
      "building tree 143 of 200\n",
      "building tree 144 of 200\n",
      "building tree 145 of 200\n",
      "building tree 146 of 200\n",
      "building tree 147 of 200\n",
      "building tree 148 of 200\n",
      "building tree 149 of 200\n",
      "building tree 150 of 200\n",
      "building tree 151 of 200\n",
      "building tree 152 of 200\n",
      "building tree 153 of 200\n",
      "building tree 154 of 200\n",
      "building tree 155 of 200\n",
      "building tree 156 of 200\n",
      "building tree 157 of 200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done 154 tasks      | elapsed:  7.8min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "building tree 158 of 200\n",
      "building tree 159 of 200\n",
      "building tree 160 of 200\n",
      "building tree 161 of 200\n",
      "building tree 162 of 200\n",
      "building tree 163 of 200\n",
      "building tree 164 of 200\n",
      "building tree 165 of 200\n",
      "building tree 166 of 200\n",
      "building tree 167 of 200\n",
      "building tree 168 of 200\n",
      "building tree 169 of 200\n",
      "building tree 170 of 200\n",
      "building tree 171 of 200\n",
      "building tree 172 of 200\n",
      "building tree 173 of 200\n",
      "building tree 174 of 200\n",
      "building tree 175 of 200\n",
      "building tree 176 of 200\n",
      "building tree 177 of 200\n",
      "building tree 178 of 200\n",
      "building tree 179 of 200\n",
      "building tree 180 of 200\n",
      "building tree 181 of 200\n",
      "building tree 182 of 200\n",
      "building tree 183 of 200\n",
      "building tree 184 of 200\n",
      "building tree 185 of 200\n",
      "building tree 186 of 200\n",
      "building tree 187 of 200\n",
      "building tree 188 of 200\n",
      "building tree 189 of 200\n",
      "building tree 190 of 200\n",
      "building tree 191 of 200\n",
      "building tree 192 of 200\n",
      "building tree 193 of 200\n",
      "building tree 194 of 200\n",
      "building tree 195 of 200\n",
      "building tree 196 of 200\n",
      "building tree 197 of 200\n",
      "building tree 198 of 200\n",
      "building tree 199 of 200\n",
      "building tree 200 of 200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done 200 out of 200 | elapsed: 10.2min finished\n",
      "[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  33 tasks      | elapsed:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 0.011994223876487058\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done 154 tasks      | elapsed:    0.4s\n",
      "[Parallel(n_jobs=4)]: Done 200 out of 200 | elapsed:    0.5s finished\n"
     ]
    }
   ],
   "source": [
    "model2_2 = RandomForestRegressor(n_estimators=200,oob_score=True,verbose=2,n_jobs=4)\n",
    "#训练集拟合 \n",
    "model2_2.fit(X_train, y_train)\n",
    "# model2_2= joblib.load(\"model2_2.m\")   #如果已经保存模型，可以将上述两行注释掉，直接运行此行读取model2_2  之后类似处同理\n",
    "#进行预测\n",
    "preds2_2 = model2_2.predict(X_valid)\n",
    "#模型评价\n",
    "# score2_2 = mean_absolute_error(y_valid, preds2_2)\n",
    "score2_2 = mean_squared_error(y_valid, preds2_2)    #测量均方误差MSE\n",
    "# print('MAE:', score2_2) \n",
    "print('MSE:', score2_2) #0.011994223876487058"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['model2_2.m']"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# joblib.dump(clf, \"train_model.m\") #存储\n",
    "# clf = joblib.load(\"train_model.m\") #调用\n",
    "joblib.dump(model2_2, \"model2_2.m\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.XGBoost模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 XGBoost模型-100个预测器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17:03:36] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 70 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:37] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 84 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:37] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 90 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:38] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 106 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:38] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 106 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:39] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 100 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:39] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 110 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:40] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:40] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:40] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:41] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:41] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:42] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:42] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 120 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:43] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 120 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:43] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:44] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 114 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:44] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 114 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:45] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 126 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:45] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:46] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:46] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 112 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:47] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 126 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:47] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 102 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:48] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 110 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:48] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:49] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:49] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 98 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:50] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 110 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:50] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:51] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:51] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 62 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:51] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 102 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:52] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 94 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:52] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 102 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:53] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:53] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 110 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:54] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 90 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:54] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:55] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 90 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:55] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:56] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 102 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:56] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:57] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 68 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:58] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:03:59] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 100 extra nodes, 0 pruned nodes, max_depth=6\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception ignored on calling ctypes callback function: <function _log_callback at 0x000001FCB3247820>\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\ProgramData\\Anaconda3\\lib\\site-packages\\xgboost\\core.py\", line 119, in _log_callback\n",
      "    def _log_callback(msg):\n",
      "KeyboardInterrupt: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17:04:00] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 90 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:00] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 86 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:01] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:02] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 46 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:02] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 92 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:03] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 112 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:03] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 126 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:04] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 112 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:05] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 108 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:05] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 126 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:06] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 112 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:07] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 106 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:07] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 80 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:08] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 114 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:09] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 80 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:09] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:10] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 46 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:10] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 104 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:11] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 102 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:12] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 88 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:12] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 66 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:13] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:14] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 120 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:15] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 92 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:15] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:16] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:17] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 126 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:17] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 94 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:18] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 116 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:19] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:19] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:20] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 126 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:21] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 112 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:21] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 108 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:22] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 86 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:22] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 104 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:23] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 58 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:24] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 112 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:24] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:25] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 124 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:25] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 96 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:26] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:27] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 64 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:27] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 122 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:28] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 94 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:28] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 114 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:29] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 78 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:30] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 90 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:30] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 100 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:31] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 90 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:31] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 108 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:32] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 62 extra nodes, 0 pruned nodes, max_depth=6\n",
      "[17:04:33] INFO: C:/Users/Administrator/workspace/xgboost-win64_release_1.3.0/src/tree/updater_prune.cc:101: tree pruning end, 118 extra nodes, 0 pruned nodes, max_depth=6\n",
      "MSE: 0.010711505144585076\n"
     ]
    }
   ],
   "source": [
    "#XGBRegressor模型   参考文档 https://xgboost.readthedocs.io/en/latest/parameter.html#general-parameters\n",
    "# XGBClassifier(learning_rate =0.1,n_estimators=1000,max_depth=4,min_child_weight=6,gamma=0,subsample=0.8,colsample_bytree=0.8,reg_alpha=0.005,objective= 'binary:logistic',\n",
    "#  nthread=4,scale_pos_weight=1,seed=27)   #XGBoost的可调参数\n",
    "model3_1 = XGBRegressor(verbosity=2)   #默认n_estimators=100,XGBoost 默认开启并行，且进程数设置为按当前机器配置的最大核心数\n",
    "model3_1.fit(X_train, y_train)\n",
    "# model3_1 = joblib.load(\"model3_1.joblib.dat\")\n",
    "#进行预测\n",
    "preds3_1 = model3_1.predict(X_valid)\n",
    "#模型评价\n",
    "# score3_1 = mean_absolute_error(y_valid, preds3_1)\n",
    "score3_1= mean_squared_error(y_valid, preds3_1)    #测量均方误差MSE\n",
    "# print('MAE:', score2_2) \n",
    "print('MSE:', score3_1) #0.010711505144585076"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joblib.dump(model3_1, \"model3_1.joblib.dat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 XGBoost模型-800个预测器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MSE: 0.007166229080095838\n"
     ]
    }
   ],
   "source": [
    "# help(XGBRegressor)\n",
    "model3_2 = XGBRegressor(n_estimators=800,verbosity=2)   #默认n_estimators=500   设置verbosity=1，=0不显示任何输出，看不到进度...\n",
    "model3_2.fit(X_train, y_train)\n",
    "# model3_2 = joblib.load(\"model3_2.joblib.dat\")\n",
    "#进行预测\n",
    "preds3_2 = model3_2.predict(X_valid)\n",
    "#模型评价\n",
    "score3_2= mean_squared_error(y_valid, preds3_2)    #测量均方误差MSE\n",
    "# print('MAE:', score2_2) \n",
    "print('MSE:', score3_2)     #0.007166229080095838\n",
    "#速度较慢，但mse确实有一定降低"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['model3_2.joblib.dat']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(model3_2, \"model3_2.joblib.dat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目前来看XGBoost模型效果最好，可以进一步考虑调整树深度max_depth等参数，对于SVM方法，样本太多，速度极慢，不太适合，已弃用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.深度森林"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2021-05-04 09:25:40.219] Start to fit the model:\n",
      "[2021-05-04 09:25:40.268] Fitting cascade layer = 0 \n",
      "[2021-05-04 09:26:46.639] layer = 0  | Val MSE = 1.53138 | Elapsed = 66.371 s\n",
      "[2021-05-04 09:26:46.768] Fitting cascade layer = 1 \n",
      "[2021-05-04 09:27:56.130] layer = 1  | Val MSE = 0.24878 | Elapsed = 69.361 s\n",
      "[2021-05-04 09:27:56.239] Fitting cascade layer = 2 \n",
      "[2021-05-04 09:29:02.009] layer = 2  | Val MSE = 0.15105 | Elapsed = 65.768 s\n",
      "[2021-05-04 09:29:02.092] Fitting cascade layer = 3 \n",
      "[2021-05-04 09:30:14.509] layer = 3  | Val MSE = 0.12453 | Elapsed = 72.417 s\n",
      "[2021-05-04 09:30:14.607] Fitting cascade layer = 4 \n",
      "[2021-05-04 09:31:26.206] layer = 4  | Val MSE = 0.11913 | Elapsed = 71.599 s\n",
      "[2021-05-04 09:31:26.295] Fitting cascade layer = 5 \n",
      "[2021-05-04 09:32:33.709] layer = 5  | Val MSE = 0.10763 | Elapsed = 67.414 s\n",
      "[2021-05-04 09:32:33.785] Fitting cascade layer = 6 \n",
      "[2021-05-04 09:33:42.028] layer = 6  | Val MSE = 0.10991 | Elapsed = 68.242 s\n",
      "[2021-05-04 09:33:42.035] Early stopping counter: 1 out of 2\n",
      "[2021-05-04 09:33:42.124] Fitting cascade layer = 7 \n",
      "[2021-05-04 09:34:46.810] layer = 7  | Val MSE = 0.10812 | Elapsed = 64.687 s\n",
      "[2021-05-04 09:34:46.812] Early stopping counter: 2 out of 2\n",
      "[2021-05-04 09:34:46.812] Handling early stopping\n",
      "[2021-05-04 09:34:47.912] The optimal number of layers: 6\n",
      "[2021-05-04 09:34:50.088] Start to evalute the model:\n",
      "[2021-05-04 09:34:52.526] Evaluating cascade layer = 0 \n",
      "[2021-05-04 09:35:38.104] Evaluating cascade layer = 1 \n",
      "[2021-05-04 09:36:11.605] Evaluating cascade layer = 2 \n",
      "[2021-05-04 09:37:04.170] Evaluating cascade layer = 3 \n",
      "[2021-05-04 09:37:21.474] Evaluating cascade layer = 4 \n",
      "[2021-05-04 09:37:33.651] Evaluating cascade layer = 5 \n",
      "MSE: 0.08480740421585917\n"
     ]
    }
   ],
   "source": [
    "#深度森林模型参数较少，而且对参数不太敏感，这里只测试一次   参考文档https://deep-forest.readthedocs.io/en/latest/\n",
    "model4 = CascadeForestRegressor(verbose=1,n_jobs=4)\n",
    "model4.fit(X_train, y_train)\n",
    "# joblib.load('model4.sav')\n",
    "#进行预测\n",
    "preds4 = model4.predict(X_valid)\n",
    "#模型评价\n",
    "score4= mean_squared_error(y_valid, preds4)    #测量均方误差MSE\n",
    "print('MSE:', score4)  #0.08480740421585917"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['model4.sav']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(model4, \"model4.sav\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5.神经网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 1, loss = 35.35996983\n",
      "Iteration 2, loss = 15.85158197\n",
      "Iteration 3, loss = 14.12517475\n",
      "Iteration 4, loss = 13.16111343\n",
      "Iteration 5, loss = 12.37274687\n",
      "Iteration 6, loss = 11.75045215\n",
      "Iteration 7, loss = 11.30214957\n",
      "Iteration 8, loss = 10.80774367\n",
      "Iteration 9, loss = 10.55445653\n",
      "Iteration 10, loss = 10.19711698\n",
      "Iteration 11, loss = 9.84135670\n",
      "Iteration 12, loss = 9.68688114\n",
      "Iteration 13, loss = 9.42127557\n",
      "Iteration 14, loss = 9.28647806\n",
      "Iteration 15, loss = 9.05609412\n",
      "Iteration 16, loss = 8.94975743\n",
      "Iteration 17, loss = 8.79938648\n",
      "Iteration 18, loss = 8.51948480\n",
      "Iteration 19, loss = 8.51838847\n",
      "Iteration 20, loss = 8.67299633\n",
      "Iteration 21, loss = 8.30427131\n",
      "Iteration 22, loss = 7.98513227\n",
      "Iteration 23, loss = 7.95501387\n",
      "Iteration 24, loss = 7.93283340\n",
      "Iteration 25, loss = 8.21421613\n",
      "Iteration 26, loss = 7.69117955\n",
      "Iteration 27, loss = 7.67580813\n",
      "Iteration 28, loss = 7.53253100\n",
      "Iteration 29, loss = 7.52780468\n",
      "Iteration 30, loss = 7.51329036\n",
      "Iteration 31, loss = 7.50668554\n",
      "Iteration 32, loss = 7.23770161\n",
      "Iteration 33, loss = 7.42468323\n",
      "Iteration 34, loss = 7.16870262\n",
      "Iteration 35, loss = 7.20259127\n",
      "Iteration 36, loss = 7.04838615\n",
      "Iteration 37, loss = 7.13795724\n",
      "Iteration 38, loss = 7.00597181\n",
      "Iteration 39, loss = 6.87092318\n",
      "Iteration 40, loss = 6.85024063\n",
      "Iteration 41, loss = 7.12678247\n",
      "Iteration 42, loss = 6.74364163\n",
      "Iteration 43, loss = 6.64623142\n",
      "Iteration 44, loss = 6.53376546\n",
      "Iteration 45, loss = 6.93684628\n",
      "Iteration 46, loss = 6.73955804\n",
      "Iteration 47, loss = 6.53519143\n",
      "Iteration 48, loss = 6.54653983\n",
      "Iteration 49, loss = 6.42828967\n",
      "Iteration 50, loss = 6.45046770\n",
      "Iteration 51, loss = 6.62386445\n",
      "Iteration 52, loss = 6.48123809\n",
      "Iteration 53, loss = 6.21704389\n",
      "Iteration 54, loss = 6.35012212\n",
      "Iteration 55, loss = 6.27060824\n",
      "Iteration 56, loss = 6.36101567\n",
      "Iteration 57, loss = 6.55215349\n",
      "Iteration 58, loss = 6.03969865\n",
      "Iteration 59, loss = 5.99605637\n",
      "Iteration 60, loss = 6.26239089\n",
      "Iteration 61, loss = 6.19399962\n",
      "Iteration 62, loss = 6.03394747\n",
      "Iteration 63, loss = 5.96851552\n",
      "Iteration 64, loss = 6.15620938\n",
      "Iteration 65, loss = 6.39747247\n",
      "Iteration 66, loss = 5.86949008\n",
      "Iteration 67, loss = 5.74975474\n",
      "Iteration 68, loss = 5.87724755\n",
      "Iteration 69, loss = 5.89752262\n",
      "Iteration 70, loss = 6.19418218\n",
      "Iteration 71, loss = 5.87208937\n",
      "Iteration 72, loss = 5.92361908\n",
      "Iteration 73, loss = 6.03686051\n",
      "Iteration 74, loss = 5.79467783\n",
      "Iteration 75, loss = 5.63089837\n",
      "Iteration 76, loss = 5.81661946\n",
      "Iteration 77, loss = 5.77126420\n",
      "Iteration 78, loss = 5.79824695\n",
      "Iteration 79, loss = 5.64217352\n",
      "Iteration 80, loss = 6.04018406\n",
      "Iteration 81, loss = 5.73251110\n",
      "Iteration 82, loss = 5.60556365\n",
      "Iteration 83, loss = 5.54319762\n",
      "Iteration 84, loss = 5.71868842\n",
      "Iteration 85, loss = 5.73166658\n",
      "Iteration 86, loss = 5.60545625\n",
      "Iteration 87, loss = 5.74517190\n",
      "Iteration 88, loss = 5.49757007\n",
      "Iteration 89, loss = 5.83918461\n",
      "Iteration 90, loss = 5.36499081\n",
      "Iteration 91, loss = 5.53497405\n",
      "Iteration 92, loss = 5.48343332\n",
      "Iteration 93, loss = 5.41994884\n",
      "Iteration 94, loss = 5.62172477\n",
      "Iteration 95, loss = 5.69372273\n",
      "Iteration 96, loss = 5.43285767\n",
      "Iteration 97, loss = 5.23595354\n",
      "Iteration 98, loss = 5.26591612\n",
      "Iteration 99, loss = 5.38271025\n",
      "Iteration 100, loss = 5.63635409\n",
      "Iteration 101, loss = 5.43703202\n",
      "Iteration 102, loss = 5.35214202\n",
      "Iteration 103, loss = 5.51596148\n",
      "Iteration 104, loss = 5.31615689\n",
      "Iteration 105, loss = 5.32428430\n",
      "Iteration 106, loss = 5.42591465\n",
      "Iteration 107, loss = 5.24959732\n",
      "Iteration 108, loss = 5.25785429\n",
      "Training loss did not improve more than tol=0.000100 for 10 consecutive epochs. Stopping.\n",
      "MSE: 17.83978465940722\n"
     ]
    }
   ],
   "source": [
    "#失败品，MSE明显过大\n",
    "model5= MLPRegressor(hidden_layer_sizes=300,max_iter=500,verbose=True)    #默认隐藏层100\n",
    "model5.fit(X_train, y_train)\n",
    "# model5= joblib.load(\"model5.m\")\n",
    "#进行预测\n",
    "preds5 = model5.predict(X_valid)\n",
    "#模型评价\n",
    "score5= mean_squared_error(y_valid, preds5)    #测量均方误差MSE \n",
    "print('MSE:', score5) #17.83978465940722"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['model5.m']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(model5, \"model5.m\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE: 3.5145003916485815\n"
     ]
    }
   ],
   "source": [
    "#KNN回归 失败品 MAE明显过大\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "knn = KNeighborsRegressor(n_neighbors=10)\n",
    "my_pipeline_knn = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                              ('model', knn)\n",
    "                             ])\n",
    "# Preprocessing of training data, fit model \n",
    "my_pipeline_knn.fit(X_train, y_train)\n",
    "\n",
    "# Preprocessing of validation data, get predictions\n",
    "preds_knn = my_pipeline_knn.predict(X_valid)\n",
    "\n",
    "# Evaluate the model\n",
    "score = mean_absolute_error(y_valid, preds_knn)\n",
    "print('MAE:', score)    #MAE 3.5145  耗时一节课"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc-autonumbering": false,
  "toc-showcode": true,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
